User Guide-Custom Application Development

Table of Contents [Sticky]

Custom Application Development

PVA ISP provides a C++ library for constructing image-processing graphs on NVIDIA Jetson's Programmable Vision Accelerator (PVA). An application chooses stages, connects their pads, sets properties, binds input and output frames, and runs the pipeline. The examples isp_minimal, isp_app, isp_minimal_cuda, and isp_multicamera show progressively more involved integrations. The API declarations are under include/pva_isp/core/; stage implementations are under src/core/stages/.

The YAML file used by the examples is an application-level configuration format. Applications calling the library directly can set stage properties through Pipeline::setStageProperty() without a YAML loader.

PVA ISP Architecture

A Pipeline owns its stage instances and registered frame descriptors. The application constructs the graph before execution. Each stage has named input and output pads; ordinary stages use input and output, while stages that exchange statistics also use pads such as gains, scene_key, and controls where supported. link() or linkMany() adds structural connections. Format negotiation and internal frame binding happen later.

The pipeline executes stages in the order they were added. Add producer stages before their consumers and connect side-channel pads explicitly. The figures illustrate concepts; actual stage choices depend on the graph selected by the application.

Common graph shapes

A separate-stage image-to-NV12 path used by isp_minimal is:

Decompand → Black Level / WB Gains → White Balance → Demosaic
→ [Convolution] → Color Correction → Global Tone Mapping
→ Gamma Tone Mapping → NV12 Conversion

The alternative joined path is Input Stage → Pre RGB → [Convolution + Color Correction / Color Correction] → Post RGB. The tone-mapping stage can be the standard implementation or the DLUT alternative (GlobalTMDLUTStage in the separate path; PostRGBDLUTStage in the joined path). The Histogram and Auto Exposure stages can form another branch when sensor control is needed. Stage formats and parameters are documented in Supported ISP algorithms.

Core Concepts

API type or concept	Role
`isp::FrameSpec`	Describes width, height, format, data type, row stride in elements, bit depth, and total buffer size in bytes.
`isp::Frame`	Holds a `FrameSpec`, device pointer, host pointer, and storage ownership information. Frames on ordinary graph links are created by the pipeline during negotiation.
`isp::Stage`	Base class for processing blocks. A stage validates pad formats, binds frames, configures resources, processes one pass, and exposes properties.
Pad	A named stage input or output, such as `input`, `output`, `gains`, or `scene_key`. Pads are names on a stage, not a separate public Pad class.
Link	A pipeline-owned connection between an output pad and an input pad. `Pipeline::Link` is private implementation data, not an application-instantiated class.
`isp::Pipeline`	Owns the stage graph, performs negotiation and allocation, configures stages, and executes them.
`isp::ExecutionSession`	Owns shared CUPVA execution resources. A pipeline creates a private session unless one is supplied before initialization. Pipelines sharing one session serialize `process()` calls.

The base Stage supports an input pad and an output pad. Stages with auxiliary data override pad methods. For example, Black Level Correction or WB Gains may output a gains frame; White Balance consumes it. Demosaic or Pre RGB may output a scene_key frame; Global Tone Mapping or Post RGB consumes it. These side frames are negotiated and bound like image frames. Optional side pads often require a property to be set before prepare(); for example, enabling compute_scene_key selects a different stage executable.

Pipeline Lifecycle

A typical integration follows these steps:

Create an isp::Pipeline and add built-in stages with addStage(StageType, name), or transfer a custom std::unique_ptr<Stage> with add().
Connect stages with link() or linkMany(). Use the pad-name overload for auxiliary data.
Set required stage properties, including any property that selects a different kernel or pad layout.
Bind boundary frames with setInput() and setOutput(). These helpers create Frame descriptors and record the stage/pad ownership. addFrame(FrameSpec) is also available for frames that need direct registration; it takes a spec, not a name.
Call prepare(enableStats). This is equivalent to init(enableStats), negotiate(), allocate(), and configure() in sequence. Check its Boolean result and lastError().
Fill the input frame's host mapping, call process(), and use the returned output host pointer or the bound output frame. process() returns nullptr on failure.

link() records topology; it does not allocate an image buffer immediately. During negotiate(), the pipeline asks upstream stages to propose frame formats, checks downstream acceptance, creates internal frames, and binds them to both ends. During allocate(), it allocates storage for pipeline-owned frames. During configure(), each stage prepares its PVA program. Runtime processing then follows stage-add order.

For multiple pipelines, an application may provide a shared ExecutionSession with setExecutionSession() before preparation. The session's profiling setting must match prepare(enableStats). Pipelines sharing a session run serially through that session; independent sessions can use independent streams, as illustrated by isp_multicamera.

Building ISP Pipelines

A complete Bayer-to-NV12 application must include the stages and side-channel connections its selected algorithm path requires. Smaller graphs can end at an intermediate format. For example, a Decompand → Demosaic graph produces planar RGB16 for an application that needs linear RGB instead of NV12. A standalone White Balance stage requires either manual gains or a connected generated-gains frame. A Global Tone Mapping stage requires a scene_key frame from an upstream stage. A Histogram → Auto Exposure branch can generate sensor exposure and gain controls. The selected FrameSpec must match every boundary stage's format, data type, stride, and size requirements.

Buffer ownership and external CUDA frames

Data	Ownership and behavior
Pipeline-owned frames	With `FrameMemoryType::kOwned`, the pipeline allocates registered boundary and internal link storage and provides host/device mappings.
External CUDA boundary frames	The application owns the allocation. Bind it through `setInput()` or `setOutput()` with `FrameMemoryType::kExternalCUDA` so the pipeline records boundary metadata and registers it with CUPVA during processing.
Stage assets and persistent state	LUTs, command programs, and temporal state are managed by the relevant stages. They are not graph links that an application must create.
Scratch buffers	Stage-local temporary resources used while processing; their layout is implementation-specific.

An external CUDA frame can be rebound between process() calls with updateExternalFrame(frame, devicePtr, hostPtr) only when its boundary stage/pad supports rebinding. The replacement storage must have matching size, memory kind, mapping, and CUPVA metadata. Its dimensions, format, stride, and bit depth remain fixed. Do not attach external CUDA storage directly through Stage::bindInput() or Stage::bindOutput(); that bypasses the boundary metadata required by the pipeline. The isp_minimal_cuda example shows input and output rebinding.

Minimal C++ Integration Example

The following function processes one left-aligned 12-bit BGGR Bayer frame into planar RGB16. It uses pipeline-owned boundary frames, configures a two-point decompand curve, and performs one pass. The input and output arrays must have the sizes stated in the function comments. A product pipeline should choose its curve and stages for the actual camera.

#include <cstddef>
#include <cstdint>
#include <cstring>
#include <vector>

#include "pva_isp/core/pipeline.h"

// raw: width * height uint16_t samples, with four zero low bits.
// rgb: 3 * width * height uint16_t samples, planar R/G/B order.
bool convertFrame(const uint16_t* raw, uint16_t* rgb,
                  uint32_t width, uint32_t height) {
  if (!raw || !rgb || width == 0 || height == 0 ||
      (width % 2) != 0 || (height % 2) != 0) return false;

  const size_t pixels = static_cast<size_t>(width) * height;
  const size_t rawBytes = pixels * sizeof(uint16_t);
  const size_t rgbBytes = 3 * rawBytes;

  isp::Pipeline pipeline;
  auto* decompand = pipeline.addStage(isp::StageType::kDecompand, "decompand0");
  auto* demosaic = pipeline.addStage(isp::StageType::kDemosaic, "demosaic0");
  if (!decompand || !demosaic || !pipeline.link(decompand, demosaic))
    return false;

  if (!pipeline.setStageProperty("decompand0", "shift_bits", int64_t{4}) ||
      !pipeline.setStageProperty("decompand0", "x_pts",
          isp::StagePropertyValue{std::vector<uint16_t>{0, 4095}}) ||
      !pipeline.setStageProperty("decompand0", "y_pts",
          isp::StagePropertyValue{std::vector<uint16_t>{0, 65535}}))
    return false;

  const isp::FrameSpec inputSpec{
      width, height, isp::FrameFormat::kBayerBGGR,
      isp::FrameDataType::kUInt16, width, 12, rawBytes};
  const isp::FrameSpec outputSpec{
      width, height, isp::FrameFormat::kRGB,
      isp::FrameDataType::kUInt16, width, 16, rgbBytes};
  auto* input = pipeline.setInput(decompand, inputSpec);
  auto* output = pipeline.setOutput(demosaic, outputSpec);
  if (!input || !output || !pipeline.prepare(false)) return false;
  if (!input->hostPtr || !output->hostPtr) return false;

  std::memcpy(input->hostPtr, raw, rawBytes);
  const uint8_t* result = pipeline.process();
  if (!result) return false;
  std::memcpy(rgb, result, rgbBytes);
  return true;
}

The two-point curve above maps 12-bit codes to a 16-bit range; it is an integration example, not sensor calibration. prepare(false) disables per-stage statistics. For diagnostics, report pipeline.lastError() when a call fails. To build a full NV12 path, follow examples/isp_minimal.cpp for stage properties, image format, scene-key links, and output allocation.

Available Public API

The principal public headers are pipeline.h, frame.h, stage.h, stage_factory.h, stage_properties.h, and execution_session.h under include/pva_isp/core/. The built-in StageType values available through addStage() are:

Group	Built-in types
RAW input and statistics	`kDecompand`, `kBlackLevel`, `kWBGains`, `kHistogram`, `kAutoExposure`, `kInputStage`
Bayer to RGB	`kWhiteBalance`, `kDemosaic`, `kPreRGB`
RGB processing	`kConvolution`, `kColorCorrection`, `kConvolutionColorCorrection`
Output processing	`kGlobalToneMapping`, `kGlobalTMDLUT`, `kGammaToneMapping`, `kNV12Conversion`, `kPostRGB`, `kPostRGBDLUT`

StageType::kLast is a sentinel, not a creatable stage. Concrete built-in stage headers live under src/core/stages/; applications using the installed library should normally use the public StageType factory interface. The library's Pipeline::visualize() can describe a prepared graph in Graphviz DOT form. A logger can be installed with setLogger() to receive pipeline messages.

❯

Share This Page

Send Email Using